Preliminary System Design for an EDA Assistant

Data analysis plays a central role in our attempts to understand the behavior of complex systems. While research in both statistics and arti cial intelligence has addressed issues in the automation of later stages of analysis, such as theory generation, model selection, and experiment design [7], less attention has been given to initial exploration of data. Deriving structure from data is nevertheless a necessary rst step. Exploratory data analysis (EDA) [8] provides a wide range of statistical tools for looking at data. Human analysts nd it straightforward to select and apply these tools e ectively. The di culty in automating exploration is one of control. At any point we can apply a large number of complex functional transformations or clustering operations to the data; to each result we can apply the same set of operations. The problem grows explosively. We have developed a blackboard-based design of an automated system that acts as an EDA assistant to a human analyst. The system maintains a sharp boundary between data-directed and goal-directed exploration. Data-directed mechanisms extract simple observations and suggestive indications from the data. EDA operations are then applied in goal-directed fashion to generate deeper descriptions of the data. Control rules guide the EDA operations, relying on intermediate results for their decisions. This abstract gives an overview of the design, which is partly implemented. We emphasize that the work is incomplete.